[Issue #799] Transform PoC by jcrichlake · Pull Request #810 · HHS/simpler-grants-protocol

jcrichlake · 2026-05-07T20:07:57Z

Summary

Fixes [SDK] Draft transforms PoC: Python #799
Time to review: 15 minutes

Changes proposed

Implements a bidirectional transform framework as a proof-of-concept for the plugin system. Plugin authors can now define declarative mapping dicts that compile into to_common /
from_common callables, enabling data translation between a source system's native format and the CommonGrants format without writing imperative transform code.

New utilities:

build_transforms() — compiles a pair of mapping dicts into typed (to_common, from_common) callables with structural validation at call time
TransformResult — unconditional return shape carrying a result dict and a list of PluginErrors (non-fatal; partial result always returned)
PluginError — structured error type with path, handler, source_value, and cause fields for programmatic error handling
ObjectSchemasInput — bundles to_common / from_common for a single object type; passed to define_plugin() via transform_schemas
PluginMeta — optional metadata attached to a plugin (name, version, source_system, capabilities)

Built-in mapping handlers (in utils/transformation.py):

Handler	Description
`field`	Extracts a value via dot-notation path
`const`	Returns a fixed literal value
`match`	Case-based lookup (canonical ADR name)
`switch`	Alias for `match` (backward compat)
`numberToString`	Extracts numeric value, coerces to string
`stringToNumber`	Extracts string value, coerces to int/float

Custom handlers can be registered per build_transforms() call and cannot override built-in names.

Sample grants.gov plugin (examples/plugins/grants_gov/): demonstrates the full interface — bidirectional field mapping, status case conversion, funding amounts, key dates, and
plugin metadata.

Documentation (extensions/README.md): new "Bidirectional Transforms" section covering mapping format, handler reference, custom handlers, and roundtrip usage.

Tests (tests/extensions/, tests/utils/): coverage for build_transforms, TransformResult, PluginError, ObjectSchemasInput, PluginMeta, and transform_from_mapping.

Context for reviewers

This is a proof-of-concept scoped to the transform layer only. It is intentionally limited:

build_transforms() validates mapping structure at call time but cannot validate that field paths resolve against real data (deferred to full SDK with schema introspection).
The PoC does not auto-invert one mapping from the other — both directions must be authored explicitly, because handlers like match are not reversible.
The TODO (full SDK) comments in transforms.py call out where model_validate integration belongs (in define_plugin(), which knows the target Pydantic model).

Verification:

Run the working example end-to-end:

cd lib/python-sdk
poetry run python examples/transforms.py

This executes a grants.gov → CommonGrants → grants.gov roundtrip and prints a PASS/FAIL check for each mapped field.

Run the test suite:
cd lib/python-sdk
poetry run pytest tests/extensions/ tests/utils/ -v

Additional information

Sample output from examples/transforms.py:

============================================================
SOURCE DATA (grants.gov format)
============================================================
{ "data": { "opportunity_title": "Research into conservation techniques", "opportunity_status": "posted", ... } }

============================================================
to_common: grants.gov → CommonGrants
============================================================
Errors: none

Result:
{ "title": "Research into conservation techniques", "status": { "value": "open", ... }, "funding": { "minAwardAmount": { "amount": 10000, "currency": "USD" }, ... } }

============================================================
from_common: CommonGrants → grants.gov
============================================================
Errors: none

============================================================
ROUNDTRIP CHECK
============================================================
  [PASS] title: 'Research into conservation techniques' -> 'Research into conservation techniques'
  [PASS] status: 'posted' -> 'posted'
  [PASS] award_floor: 10000 -> 10000
  [PASS] award_ceiling: 100000 -> 100000

Roundtrip result: ALL PASS

jcrichlake · 2026-05-08T17:51:44Z

+    def to_common(native: Any) -> TransformResult[Any]:
+        try:
+            result = transform_from_mapping(native, to_common_mapping, handlers=merged)
+            # TODO (full SDK): run model_validate on result and append validation


I'm not sure if we should handle this TODO here or later down the road. Would love thoughts on this.

github-actions · 2026-05-08T19:19:34Z

🚀 Website Preview Deployed!

Preview your changes at: https://cg-pr-810.billy-daly.workers.dev

This preview will be automatically deleted when the PR is closed.

jcrichlake · 2026-05-08T19:27:52Z

+    to_common_mapping: dict[str, Any],
+    from_common_mapping: dict[str, Any],
+    handlers: dict[str, Handler] | None = None,
+    common_model: type[BaseModel] | None = None,


Calling this out as a needed change from the existing ADR. Without having the common_model in the signature we don't have a way to validate an extended class that contains optional fields. @SnowboardTechie for awareness

SnowboardTechie

Nice work on round 2. 7 of my 9 earlier findings are fully addressed, and the other 2 are documented-as-intended. The HandlerError refactor in particular reads much cleaner than the hardcoded handler=None we started with.

One latent bug worth surfacing before this ships: transform_node silently treats handler keys as literals when a non-handler sibling iterates first. The fact that build_transforms and transform_from_mapping are both public surface now makes it more load-bearing than before. Reproducer in the inline on utils/transformation.py.

Two small follow-ups on the round-2 fix itself, both flagged on individual threads: HandlerError is a behavior change for dump_with_mapping / validate_with_mapping callers (worth deciding whether to make it ValueError-compatible), and the new attribution path is not covered by an assertion in the exception test.

SnowboardTechie

Round-3 looks great! The ADR has some drift against #803 in the merge_extensions / filters area, but IMO we should rebase main onto current HOLD-transforms after this merges.

Approving w/ 2 small suggests: git rm lib/python-sdk/.coverage and add .coverage to either lib/python-sdk/.gitignore or the root. It looks like Pytest-cov SQLite output landed in the tree. And another trimming an extra /.

SnowboardTechie · 2026-05-12T15:57:01Z

Not sure we want this commited?

Co-authored-by: Bryan Thompson <18094023+SnowboardTechie@users.noreply.github.com>

Port the Python transforms PoC (PR #810, branch 799-transform-poc-fetch) to @common-grants/sdk so the ADR-0022 / ADR-0017 contract is validated in both SDKs before either is locked in for full implementation. Public additions under @common-grants/sdk/extensions: - buildTransforms() — compile a pair of ADR-0017 mapping objects into typed (toCommon, fromCommon) callables with call-time structural validation. Optional commonModel Zod schema turns parse failures into PluginError[] rather than thrown exceptions. - TransformResult<T> — unconditional { result, errors } return shape (ADR-0022 Decision #7). - PluginError — structured error class with path / handler / sourceValue / cause (ADR-0022 Decision #9). Docstring documents the PII surface on both sourceValue (carries the full input record) and message (data-bearing on the Zod-validation path because Zod's default error map embeds received values). - transformFromMapping(), getFromPath(), DEFAULT_HANDLERS — mapping runtime; six built-in handlers (const, field, match, switch alias, numberToString, stringToNumber). - definePlugin() accepts optional meta and transformSchemas. Existing callers passing only `extensions` are unaffected. Security hardening (mapping JSON may be reconstituted from untrusted sources via mergeExtensions(), so the runtime must fail loud on hostile shapes): - buildTransforms() rejects custom handler names that collide with the default registry or shadow Object.prototype keys (constructor, toString, __proto__, etc.) at call time. - validateMapping() rejects `__proto__` as an output field name at build time; transformFromMapping() rejects it again at walk time so the JSON attack vector (own-enumerable __proto__ key from JSON.parse) fails fast in both places. - stringToNumber's error message does not embed the source value (would flow into PluginError.message and bypass the sourceValue PII guard). Out of scope (matches Python PoC; deferred to full SDK): - Auto-generation of transforms from declarative extensions.schemas[obj].mappings inside definePlugin() (Decision #6 TODO). - Always-on commonModel validation inside definePlugin() — opt-in at buildTransforms() for now (Decision #7 TODO). Includes: - examples/transforms.ts round-trip (pnpm example:transforms) - README "Plugin transformations" section + API reference table - 5 new define-plugin specs, 30 transformation-handler specs, 12 buildTransforms specs (427 tests total, all passing) - Minor changeset bump for @common-grants/sdk Targets HOLD-transforms per the SDK Plugin Enhancements branching strategy.

github-actions · 2026-05-13T19:40:17Z

🗑️ Preview Cleaned Up

The preview for this PR has been automatically deleted.

Issue #799 Transform PoC

bf1e8a8

github-actions Bot added python Issue or PR related to Python tooling sdk Issue or PR related to our SDKs py-sdk Related to Python SDK labels May 7, 2026

jcrichlake and others added 3 commits May 7, 2026 16:22

Fixing plugin in makefile

3e8d260

Merge branch 'main' into 799-transform-poc

12bde81

Adding custom handler example

8850fb1

jcrichlake marked this pull request as ready for review May 8, 2026 16:33

jcrichlake commented May 8, 2026

View reviewed changes

Updating transform contract

c6af48c

github-actions Bot added website Issues related to the website typescript Issue or PR related to TypeScript tooling labels May 8, 2026

Fixing make:checks errors

0139cde

jcrichlake commented May 8, 2026

View reviewed changes

Reducing duplicate tests and updating makefile

bb1c766

SnowboardTechie self-requested a review May 11, 2026 18:24

SnowboardTechie reviewed May 11, 2026

View reviewed changes

jcrichlake changed the base branch from main to HOLD-transforms May 11, 2026 18:59

SnowboardTechie reviewed May 11, 2026

View reviewed changes

Comment thread lib/python-sdk/common_grants_sdk/extensions/types.py Outdated

jcrichlake added 2 commits May 11, 2026 15:46

Fixing pr comments.

84576f6

Adding ADR fixes

ffd4051

SnowboardTechie reviewed May 11, 2026

View reviewed changes

Comment thread lib/python-sdk/common_grants_sdk/utils/transformation.py

Comment thread lib/python-sdk/common_grants_sdk/utils/transformation.py Outdated

Comment thread lib/python-sdk/tests/extensions/test_transforms.py Outdated

PR comments rd 2

915c6c5

SnowboardTechie approved these changes May 12, 2026

View reviewed changes

jcrichlake and others added 2 commits May 12, 2026 13:14

Update website/src/content/docs/governance/adr/0022-plugin-framework.mdx

bea786d

Co-authored-by: Bryan Thompson <18094023+SnowboardTechie@users.noreply.github.com>

Merge branch 'HOLD-transforms' into 799-transform-poc

2d22aba

SnowboardTechie mentioned this pull request May 12, 2026

[Issue #798] Transforms PoC: TypeScript #825

Draft

jcrichlake merged commit 21c0c8e into HOLD-transforms May 13, 2026
6 checks passed

jcrichlake deleted the 799-transform-poc branch May 13, 2026 19:39

Conversation

jcrichlake commented May 7, 2026

Summary

Changes proposed

Context for reviewers

Uh oh!

jcrichlake May 8, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jcrichlake May 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SnowboardTechie left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

SnowboardTechie left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

SnowboardTechie May 12, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions Bot commented May 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented May 8, 2026 •

edited

Loading

SnowboardTechie left a comment •

edited

Loading